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ABSTRACT 



A system for rate control during transcoding and encoding of 
digital video data in a multi-program transmission environ- 
ment accommodates changes in the GOP structure of any of 
the programs, e.g., due to a commercial insertion or the like. 
A hierarchical bit allocation scheme is used, where bits are 
allocated on a super group of pictures (GOP) level, i.e., a 
grouping of multiple GOPs, a super frame level, and an 
individual frame level. The bit allocation for the super GOP 
and each picture is adjusted when a change in the structure 
of a group of pictures (GOP) is detected for any of the 
pre-compressed programs. The super GOP length may also 
be adjusted if required to maintain an integer number of 
GOPs therein. A change in the GOP structure is detected 
based on a change in the GOP length or sub_GOP length 
(distance between P pictures). For pre-compressed data, 
initial GOP lengths and sub_GOP lengths can be assumed 
and adjusted as required based on actual measurements. For 
a current I or P picture, the sub GOP length is calculated 
based on temporal references of the current picture and a 
previous I or P picture. For a current B picture, the same 
calculation is used, plus one picture. 

34 Claims, 7 Drawing Sheets 
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1 2 

STATISTICAL MULTIPLEXER AND mercial insenioo, changes in the program contenl (e.g., due 

REMULTIPLEXER THAT ACCOMMODATES to a transition from a movie to a news announcement or a 

CHANGES IN STRUCTURE OF GROUP OF sports event), and the like. The system should handle struc- 

PICTURES turc changes that occur at a GOP boundary or within a GOP. 

5 The system should accommodate changes in a GOP 
length and/or a sub_GOP length. 

BACKGROUND OF THE INVENTION ,y,^,^ ^^^jj difficulties in integrating 

The present invention relates to rate control during pre-compressed program bit streams (that include, e.g., 

transcoding and encoding of digiUl video programs in a video materials such as films and the like that are pre- 

multi-program transmission environment, where several compressed and stored for subsequent transmission) wdlh 

programs are mulUplexed and transmitted over a single uncompressed digital video sequences, 

communication channel. The system should not require a priori (beforehand) 

Commonly, it is necessary to adjust a bit rate of digital knowledge of the picture organization (GOP structure) of 

video programs that are provided, e.g., to subscriber termi- ^5 programs (e.g., the GOP length, and arrangement of different 

nals in a cable television network or the like. For example, P^^^^ ^yP^ ^OP). The system should avoid a 

a first group of signals may be received at a headend via a processing delay of about one GOP which would otherwise 

satelUte transmission. The headend operator may desire to ^e mcurred to extract the complete GOP structure informa- 

forward selected programs to the subscribers while adding ^'^^"^ » pre-compressed bit stream. The system should 

programs (e.g., commercials or other content) from a local .q ^^^^ corresponding to the 

source, such as storage media or a local live feed. ^OP, thereby reducing the memory size required for 

Additionally, it is often necessary to provide the programs transcodiDg. 

within an overall available channel bandwidth. It may also The present invention provides a system having the above 

be desired to change the relative quality level of a program ^Jid other advantages. 

by allocating more or fewer bits. ^5 SUMMARY OF THE INVENTION 

Accordingly, the statisUcal multiplexer (stat mux) or ^^^^^^ ^^^^^^^ ^^^^^^^ ^ ^^^^ ^^^^^^^ 

encoder, which mcludes a number of encoders for encoding (.^s^odiDg and coding of digital video data in a multi- 

uncompressed digital video signals at a specified bit rate, has ^ transmission environment, 

been developed. The statistical remultiplexer (stat remux), Jn.- c 

J u • u u ji -1 J A u •* The mvention accounts for changes m the structure of a 

or transcoder, which handles pre-compressed video bit 30 , , • t ■ 11 . j- . 

. . •« J u*» u GOP, e.g., due to a commercial mscrtion or the luce, to adjust 

streams by re-compressmg them at a specified bit rale, has , J. , • ^ j- 1 a i_ • 

1. J the bit allocation for each picture accordingly. A change m 

also been developed. Moreover, functions 01 a stat mux and ^^y, , ^, /• j \ sr u %r^n 

u u- J u •* - J ■ J . * A Ibe GOP length (mcrease or decrease), and/or a sub_GOP 

remux may be combmed when it is desired to transcode 1.. i_ j.jt- -""^ — . 

J J , , -1 , J J * length can be accommodated. For example, a change m 

pre-compressed data whdc also codirig uncornpressed data ^ ^ ^^^^ ibbPBBPBB . . . to IBPBPBPB . . . 

for transport m a common output bitstream. Uncompressed 35*^ . . ^^r.. c 

jjr*i.c**-* A results in the sub_GOP changing from three to two. 

programs are coded for the first time, whue compressed ^ , , . . . ^ . . 

programs are re-encoded, typically at a different bit rate. Th^ sub_GOP length contams information about the 

ir, . .^^^ ... , spcafic picture types and their arrangement m a GOP. 

For MPEG applications, a stat mux^emux mus accom- ^ ^^^^ ^ ^^ ^^^^^^ 

modate three different picture types (I P and B), which ^ ^ ^^ ^^^^ ^ ^j^^^ ^ ^ ^^^^^^^ 

u^allyrequirequitedifferentntimbersofbitsbecauseofthe 40 ^^^^^^.^t „f future types if the sub_GOP changes, 

different nature of their temporal processing. . . .. .... 

Moreover, the invention operates with a hierarchical bit 

Bit allocadon strategies should take the picture types into allocation scheme, where bits are allocated on a super GOP, 

consideration. For a given bit budget, this implies a require- {^^^^^ individual frame levels. 

•«'°^'«'ge of. Pfct^f^ organizations ^^ ^^^^^ ^ anywhere, 

(GOP) of the programs. ITie requirement is not a problem for ^^j^ ^ „^ ^ „f ^ ^Qp 

encoders because encoders can plan ahead for types and ^j,^ ^ ^ 

arrangement of pictures that it will output However, such a ^^p ^-calculated, and the allocation 

pnor. laiowledge is typically not available for transcoden;, ^ J ^^p ^ accordingly. 

which deal With pre-compressed video bit streams. . ..... ...» c u . 

50 If there is a switch during the middle of an old GOP to 

Accordingly, it would be desirable to provide a stat another program of a new GOP, this is treated as a transition 

mux/remux system that removes the assumption that the ^^^^ ^Id incomplete GOP, and then to the 

picture organizations arc available for all the programs. QQp 

The system should provide a novel adaptive bit allocation ^ particular method in accordance with the invenfion is 

strategy for a stat mux/remux system that requires no a pnon 55 provided for coding a plurality L of video programs. The 

(beforehand) knowledge of program picture organization. method includes the step of allocating bits for coding the 

The system should be able to address any changes in vijeo programs according to a hierarchical scheme that 

program picture organizations, including a change in the includes at least: (a) a super group of pictures (GOP) level, 

GOP length and/or the sub-GOP length (the distance wherein a super GOP comprises at least one GOP fi-om each 

between two P-pictures in either encoding or display order), video programs, and (b) an individual frame level. The 

The system should include transcodcrs for processing allocating step is responsive to respective GOP structures of 

pre-compressed video bit streams, or both encoders and the video programs. The respective GOP structures of the 

iranscoders, for additionally handling uncompressed digital video programs are monitored to detect any changes thereof, 

video signals. When a change in the GOP structure of at least one of the 

Moreover, the system should accommodate changes in the 65 video programs is detected, the allocation of bits for coding 

group of pictures (GOP) structure of pre-compressed the video programs in the super GOP is adjusted according 

bitstreams, for example, due to switching channels, com- to the change. 
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Accordingly, the invention can optimize the allocation of A stat mux has encoders, while a stat remux only needs 

bits since a rate control processor maintains updated knowl- transcoders, and a stat mux/remux has both encoders and 

edge of each picture type and the arrangement of pictures for transcoders, 

the different programs to the maximum extent possible. RG. 1 illustrates a stat mux/remux system in accordance 

Corresponding apparatuses are also presented. ^ with the present invention. The inputs to the stat mux/remux 

system 100 can include uncompressed digital video 

BRIEF DESCRIPTION OF THE DRAWINGS sequences and/or pre-compresscd bit streams. The uncom- 

^ .,1 . . . . / -^L i_ J pressed digital video sequences, e.g., programs 1 and 2 arc 

FIG. 1 Illustrates a slat mux/remux with both encoders ^^^^^^^ ^ ^^^^^^ and 130, respectively, e.g., which 

and transcoders, and a joml rate control engmc, m accor- ^p^^ ^.^^^ ^^^^^^^^ pre-compresscd 

dance with the present mvenuon. ^^^^^^ ^^^ ^ bitstreams) L-1 and L are processed, 

FIG. 2 illustrates a super group^f-pictures (GOP) con- respectively, by transcoders 140 and 150, GOP structure 

struct in accordance with the present invention. monitors 142 and 152, and optional bit rate functions 144 

FIG. 3 illustrates a super frame construct in accordance and 154. 

with the present invention. is Any known transcoder configuration may be used. 

FIG. 4 illustrates a decoder for receiving data from the sut The example GOP structure monitors 142 and 152 operate 

mux/remux of FIG. 1 in accordance with the present inven- as will be discussed in greater detail below to provide the 

tion. GOP lengths N^_i and N^, and sub GOP lengths Mjj,__i and 

no. 5 illustrates a video program with a change in the ^^"^ respective (L-l)th and Lth programs to the rate 

sub_GOP length, and subsequent adjustment of the super ^^^^^^^ processor UO, which adjusts the bit aUocation to the 

GOP length, in accordance v^th the present invention. ^^^^^^^ programs (via the target bit rate T) in response 

FIG 6 illustrates a video program with an increase in the "-[^e example bit rate functions 144, 154 arc optional and 

GOP length m accordance with the present mvenUon. ^^^^^^ pre-compressed input 

FIG. 7 iUaistrates a video program with a decrease m the 25 data frames. As discussed further in connection with equa- 

GOP length in accordance with the present invention. tion 14, the rate control processor may reduce the target bit 

rate for a transcoder so that it does not exceed the measured 
bit rate. 

Note that the invention is operative when only pre- 

The present invention relates to rate control during 30 compressed bitstreams are present. Uncompressed program 

transcoding and coding of digital video data in a multi- data is accommodated, but is not required. Moreover, any 

program transmission environment number of pre-compressed bitstreams may be 

accommodated, although only two are shown in FIG. 1 for 

1. INTOODUCnON simplicity. 

With recent advances in digital video compression and The transcoded data and encoded data (when present) are 

digital transmission, it is possible to deliver several video provided to a MUX 160 and buffer 170 prior to being 

programs in the same bandwidth presenUy occupied by a transmitted on a channel, typically to a decoder population, 

single analog TV channel. To be squeezed into a fixed-rate The channel may be part of a broadband communication 

channel, the video programs have to share the channel network, such as a cable or satellite television network, for 

capacity. Specifically, raw pixel data is compressed, and example. 

compressed bits are transcoded. Encoders and transcoders The pre-compressed program data may be provided from 

can be thought of as rate-conversion engines. An encoder source, such as a storage medium (e.g., magnetic tape or 

compresses a digital video sequence into a bit stream at a compact disc), or from a satellite transmission or stat mux, 

much lower rate while a transcoder converts a pre- for example. Pre-compressed data, such as for commercial 

compressed video bit stream into another bit stream at a new insertion or insertion of local news or other programming, 

(low) rate. The aggregate rate of these compressed bit "^^y be provided locally for use in conjunction with pre- 

streams, however, has to be equal to, or less than, the compressed data that is received from a remote source, 

channel rate. This can be achieved by controlling either each The buffer 170 sends a fulhiess level signal to a rale 

individual rate (independent coding) or the aggregate rate control processor 110, which may adjust the target bit 

(joint coding). allocations accordingly to avoid a buffer overflow or under- 



DETAILED DESCRIPTION OF THE 
INVENTION 



In independent coding, rate control can only be performed 



flow. 



across the Ume and spatial dimensions of a single program. ^ "f*^^ mterface 108 may communicate with the rale 

In joint coding, control is extended to the program dimen- P'^'^'^'^'^ii®'.?' '""T^!^' *° ' 

sion (e.g., multiple programs). This implies more freedom in ^""^^^'^ ^^O, 130, to designate certain encoders or 

allocating the channel capacity among programs and, " transcoders as havmg a higher pnority, and ^ 

therefore, more control of picture quality among multiple . ^^^^ mux/remux 100 provides a re aUvely uniform 

programs as well as within a program. P!^^'^ ^"^^^y designated quahty level when 

° , .u * • ui . * 1 A • * different program priontics are assigned) within a program, 

A system that IS able to put multiple video programs into j -r ^ .u- / 

„ ' , . ^ ^ ^ and if necessary, among programs. To achieve this goal, 

a nxed-rate channel is called a: ...... 60 channel capacity is dynamically distributed among pro- 

1 . stat mux if the inputs are aU uncompressed digital video gj^^jg according to a program priority as well as a frame 
signals, ^^^^2 program complexity measure. 

2. slat remux if the inputs are all pre-compressed video bit Each MPEG encoder 120, 130 or transcoder, 140, 150 
streams, or receives a target number of bits, Ti, Tj, T^_i and Tjr, 

3. ^at mux/remux if the inputs are both uncompressed 65 respectively, from a rate control processor 110 at each frame, 
digital video signals and pre-compressed video bit The rate control processor 110 includes a super GOP level 
streams. processing function 106, a super frame level processing 
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function 104, a frame level processing function 102, and a 09/141,265 to L. Wang and A. Luthra, filed Aug. 27, 1998, 
complexity processor 105. These processing functions may and entitled "Dynamic Bit Allocation For Statistical Multi- 
share common hardware such as memory and processing plexing Of Compressed And Uncompressed Digital Video 
chips, but arc shown individually for simplicity. Signals." At the top level of the hierarchy, the concept of a 

The target number of bits for each frame of a program is s super GOP (Groups of Pictures) is introduced. Specifically, 

met by adjusting the quantization parameter in the MPEG the programs are conceptually divided into super GOPs, 

encoder or transcoder. The restilting number of compressed each having the same number of I, P and B pictures, that are 

bits, R, as well as the average quantization parameter, 0, assigned the same nominal number of bits, 

used for each frame are then sent to the rate control A super frame is defined at a middle level of the hierarchy, 

processor 110 as feedback data. Specifically, the encoder lO which is a collection of frames, one from each of the 

120, encoder 130, transcoder 140 and transcoder 150 pro- programs at the same frame instant. A super frame is 

ducc Rj, Rj, R^_;i and Rjr. bits, respectively, using quanti- assigned a target rate according to its relative complexity 

zation parameters Q^, Qj, Qjf^j and Q^, re^ecAively. The measures. The same idea is extended to the regular, indi- 

complexity processor 105 calculates corresponding com- vidual frame, at a lower level of the hierarchy, where a target 

plexity values C using R and Q for each program. The rate 15 rate for a firame is proportional to its complexity measure, 

control processor 110 then determines a new target number Moreover, to ensure the encoder and decoder buffers 

of bits for each new program frame or picture based on the ^^^^^ ^^^^^^ underflow, and to limit each individual 

program complexity at the frarne level ^^^^^ additional constraints are applied on the target 

The rate control processor 110 is also responave to the ^^^^^ ^-^^ f^^^^ ^ ^ 

GOP structure data from the transcoders 140, 150. 20 

Ideally, a Stat mux/remux system distributes the channel . . .j.. ^ ^ 

capacityovertheinputprogramsaccordingtotheprograms^ ^.^h^ P^^.^'^^ hierarchical dynamic 

relative complexity measures. That is, more complex pro- ^°^^^"^^!,Sy to a slat mux/rcmux system, 

grams are assigned more bits, and less complex programs 2. 1 Super GOP and Nominal Rate 

are assigned fewer bits. For a given rate, this also implies 25 ^ 2, we conceptually dmde Lmput video 

that the rate assigned for a program depends on the com- programs mto identical super GOPs(UN) m terms of the 

plexity measures of the program and the other programs. °r °. T\ . ^'"""^'^ '^^\ r^. ^ 

Since the program complexity may vary with time, the 200 mcludes data fran^^^ 

programs* relative complexity measures vary as weU. The (^20), . . . , through program L (290). Each program 

distribution of channel capacity, or bit allocation, over 30 segment, e.g., 210, 220, 290, contains a number of comp^^ 

programs therefore has to be a time-varying function (e.g., ^^^^f °' "^T ^^T^^^T' ^ 

on a frame-by-frame basis). ^^^f ^^If ^7"' F'^^lT. u^}^^' ^'nfT^^ 

Furthermore, MPEG (ISOAEC (MPEG.2), "Generic cod- (222) . . . through program L(292). A third super GOP 204 

ing of moving pictures and associated audio", March 1994) ^^»f, ^^^f ^^^^ PJ^f^ 1 (214), program 2 

defines three picture types in terms of temporal processing 35 (^^f). • • • ; ^ou^ program L (294). 

a. P and B), and the organizaUon of the three pictures (GOP) . ^is of programs which is given and N 

can be very flexible. Due to the different namre of their f ^^""^^ ^^^/^JP^' ^n^J^^'c^u ''^''^ V^l 

temporal processing, the three pictures may require very ^^^^ ^T^T^ ^ r ^'"^"^ 

different numbers of bits. Hence, to wisely use the bit budget ^^^S^^ or N;,l-1,2, . . . , L, i.e., 

over the input programs, the program picnire organizations 40 N'LCM{n^, a^2» ■ ■ ■ » ^l) 0) 
should also be taken into consideration. However, for a 

transcoder, the program picture organization is embedded in Por example, if there are two different GOP lengths for N 

the pre-compressed bit stream and is therefore not available programs, say nine and fifteen, the super GOP length N=45. 

as the bit stream is being received. While a transcoder can Since N is the smallest integer that can be divided by all the 

learn the program picture organization of an input pre- 45 program GOP lengths, N^M, 2, L, the super GOPs(L, 

compressed bit stream, e.g., by scanning the bit stream by N) are the smallest identical groups containing the same 

about one GOP, this results in processing delays and the number of frames of each picture type. A super GOP(L,N) 

need for additional memory space, which increases costs. already contains the integer number of GOPs for each 

Moreover, a further complicaUon is that the extra memory program I, that is, N/N,. Super GOPs(L,N) with N defined in 

requirement will vary for programs with different GOP- 50 (1) contain the same number of I, P and B pictures, and 

lengths. hence, they are assigned the same nominal number of bits, 

The present invention operates on the assumption that a y. n^.}) (2\ 

priori knowledge of program GOP structtires is not available r^t^-^-dibpfi K ) 

to the transcoders. The invention starts with a reasonable where RcAawiW(w) ^ the average number of bits per frame, 

assumed program picture organizations (GOP length and 55 The number or I, P and B pictures in a GOP for program 

sub_GOP length), and provides gradual adjustments to the 1 are, respectively, N,y-1, N/^-N;/M;-N;^N;/M;-l, and 

assumed organization when necessary. A GOP length of N,^-N,-N,^-N,^N,-N;^-1. 

fifteen pictures, and a sub GOP length of three pictures is a 2.2 Super Frame and Target Rate 

reasonable assumed organization in many cases. For FIG. 3 illustrates a super frame, which is a collection of 

example, N-15 and M«3 corresponds to a picture organi- <S0 L frames, one from each of L programs at the same time 

zation of . . . BBIBBPBBPBBPBBPBBI . . . , while N«10 instant. 

and M-2 corresponds to a picture organization of . . . At each frame instant n, we can further imagine a super 

BIBPBPBPBPBI . . . frame 300, which is a collection of L frames or pictures, one 

o nvM A nAir- ui-r a t i r^r- A^^^^KI ^^o™ ^^^^ °^ ^ programs taken at the same frame instant. 

2. DYNAMIC BIT ALLOCAIION ^^^^^^^ ^^^^ ^^^^ p^^^^^^ ^ ^^^^ 

A hierarchical dynamic bit allocation strategy for a stat program 2, . . . , through frame 330 of program L are 

mux system is discussed in U.S. patent application Ser. No. conceptually arranged in the super frame 300. Clearly, a 
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super GOP consists of N super frames. Moreover, since 2.4 Target Rate for Regular Frame 

these L programs may have different GOP structures^ the L Given a target number of bits for a super frame n, the 

frames in a super frame can have different picture types. target number of bits for frame n of program 1 w 



The target number of bits for super frame n is given as: 



£- (3) 



Ufi = -; "ft. 



/=! jQ Here, the numerator on the right is the complexity mea- 

sure for frame n of program 1, and the denominator can be 

, ^ , r r c considered as the complexity measure for super frame n. 

where C,^^ is the complexity measure for frame n of ^^^^^^ ^ j^^^^ ^ ^ ^^^^ proportional to its 

program 1 with picture type t£{I,P,B}, and it can be either complexity measure. 

C,j, C,^, or C,^ depending upon the associated picture type ^ ^ j^,^ determined by equation (4) are based 

of I, P or B.TM5 defines a complexity measure for a frame „ ^^^^ j,^^^ ^^i^^;^^ complexity measnre. For 

as the product of the number of bits generated from the ,ranscoders, they may need further adjustment. SpecificaUy, 
frame and the average quantaation parameter used for the ^^ ^ ^^^^ ^ determined by (4) could be larger, 

frame. Other complexity measures are also allowed. „^ ^^^^^^^ ^^^^ ^^^^ ^^^^ •„ j ^ 

a, is a constant factor for addressing the spaUal resolution pre<ompressed bit stream. A rate -conversion transcoder is. 

of program 1. ....... , however, a device for converting a pre<ompressed video bit 

p, IS the quality (or priority) weigbtmg factor for program ^^^^^ ^^^^^^ yt stream at a new rate. It camiot 
1, which wiU be determined by the program provider. ,j, „f ^ pre-corapressed video signal 

y,^, is a constant factor used to compensate for the picture ^gardless of whether the new bit rate is smaUcr or greater 
type te{l,P,B} of frame o of program 1. It can be either y,, -fp bit rate 

or Y^. depending upon the picture type of I, P or ^ -n,^^^ allocating more bits for a (pre-compressed) frame 

n,^ o,j. and n,^ are the remaming number of I, P and B ^^^^^ ^ ^ils. Hence, in accordance with the invention, if 
pictures, respecuvely, for program 1 m the current super , , f^^^ ^ programs 1, T,^. (4) is greater 

GOP. For a new super GOP, Oiey are reset and then ^^^^ ^^j^ ^ ■ ^ ^^g.^ompressed 

decreased by one after a picture of the corresponding type is ^it stream, say R^,j^ the old rate should be appUed, i.e.. 



processed 

R is the remaining number of bits for the current super 
GOP, defined as 



(To, it < ll.u.i^ (5) 



2j " ^ This can be achieved using the example bit rale functions 

144, 154 of FIG. 1 by measuring the bit rate of a picture in 
the received bit stream, e.g., using a bit counter, and sending 

Here, T,^.j and R,_„_i are, respectively, the target and actual this information to the rate control processor 110 to reduce 

rates for frame n-1 of program 1. Hence. the allocated bits, T, for the associated transcoder, if 

required. 

L 2.5 Preventing Decoder Buffer Overflow or Underflow 

^ (Ttn-i -Ki^-i) FIG. 4 illustrates a decoder for receiving data from the stat 

'•' mux/remux of FIG. 1, where packets of a selected program 

are extracted and decoded. At the receiving end 480, the 
is the number of leftover bits from frames n-1 of program decoder 495 is allowed to select the desired program and 
lol,2, . . . , L, which can be either positive or negative. At e"'"*' (de-muluplex) the corresponding packets via a 
the beginning of a new super GOP, R needs to be updated as ^emux 485. The de-raultiplexed bit stream is at a variable 
R-R+T,,^, where R on the right side of the equation is the "^e- ^ encoder's responsibility to ensure the decoder 
number of bits leftover from the previous super GOP. which 50 overflows or underflows. AddiUonal con- 
can be positive or negative. s'fa"'-'' Iherefore applied at the encoder. 

Note that the numerator on the right side of equation (3) '^j^ can be achieved using the techniques of Uie afore- 

isthesumofcomplexitymeasuresforallthefraniesinsuper mentioned U.S. patent applicaUon Scr. No 09/553.457 to 

frame n. On the other hand, the denominator can be con- W'Y a constraint on the taiget rate for each regular frame, 

sidered as the sum of complexity measures for all the 5s 2 6- Constraint on Max and Mm Rate 

remaining frames in the current super GOP. Hence. equaUon ^e can also control the average bit rate over a certam 

(3) achially assigns a target number of bits for a super frame "^^ber of frames by limitmg the target number of bits for 

in proportion to the super frame's complexity measure. ^f""" " "Pf '«"Se- This can be achieved 

2.3. Preventing Encoder Buffer Overflow or Underflow T'lfcaaon &^"Nr09/553 457 

To prevent the encoder buffer 170 from overflowing or 60 '^^ * * * ' 

underflowing, certain constraints are applied by setting 3 CHANGES IN PROGRAM GOPS 
upper and lower bounds for the target rate for super frames, 

ITiis can be achieved according to the techniques set forth in The program GOP structure (including GOP length, type 

U.S. patent application Scr. No 09/533,457 to L. Wang, filed and arrangement of pictures, and sub GOP length) plays an 

Apr. 19, 2000, and entitled "Rate Control And Buffer 65 important role in determining a target rate for a frame, as 

Protection For Variable Bit Rate Video Programs Over A shown in equations (3,4). However, a program GOP struc- 

Constant Rate Channel". ture may vary from time to time. For example, for a 
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iranscoder, the input video bit stream may consist of seg- 
ments of dif[erent materials/content, such as films, news, Let and temporal references 
sports, music, etc., that are pre -encoded with different GOP for the previous I or P picture and the current picture for 
structures. Another example is commercial insertion, where program 1, respectively. 

the inserted commercial may be pre-coded with a different $ If the previous picture is an I or P picture, and the current 

GOP structure than the original program to be inserted. picture is a B picture, the B picture should be displayed 

In accordance with the invention, the bit allocation adapts before the I or P (in display order). Thus, the B picture has 

to changes in the program GOP stnictures, e.g., by adjusting a smaller temporal reference. Now, if the current picture is 

the size of the super GOP and other parameters for deter- an I or P picture, the actual sub_GOP length for program I 

mining the target rates of frames. jo is: 
3.1 Program sub_GOP Length 

The following shows a general form of picture organiza- M/.,^,,„rteinp_ref_pic2-temp_ref_prci. (6) 
tion for an example program 1. The first row is in display 

order and the second row is the corresponding encoding Otherwise, if the current picture is a B picture, 

15 Af,,„,^/»tcmp_ref_picl-tcmp_ief_pic2+l (7) 

Pm+2A/, • * • ^m-i-N, ' ' ' * sub_GOP is determined for a current picture whose 

. • . I«Bl(Afpi x^^m-iMi5i • ' • B„.,P„.^^„.a • • • V^^yious picture is a B-picture. The start of the current 

B«wtf_jP„*2A/. . . . L4JV • • . , . picture in FIG. 5, which is an I picture and the first picture 

Here, the subscripts are the picture's temporal reference, 20 of the GOP 552, is at a time 515. The start of the next picture 

and N, and M; arc the GOP length and sub_GOP length, is at a time 520. 

respectively. The pictures in the compressed video bit stream Clearly, if M;=M;^^„„„ the current sub_GOP for pro- 

that arc input to a transcoder are in encoding order. The gram 1 is correct. Hence, no further action is needed, 

display order is recovered at the decoder. However, if M/=iM;^^^,„„;, we set M,=M;^„^„ and adjust 

For an encoder that encodes uncompressed video data, the is other parameters as follows: 

changes in sub_GOP length of a program are available since 1. First, the program GOP length, N^, has to be divisible 

the GOP program structure is available, thus allowing the by the new M; (i.e., integrally divisible — with no 

encoder to plan ahead for bit allocation. remainder). If not, we adjust it by 

However, for a transcoder, the changes in sub„GOP 

structures are embedded in the input pre-compressed bit 30 ^n^^^iWi (8) 

stream. In accordance with the invention, the sub_GOP ^^itx^ [/j denotes integer division with truncation of the 

length of a program is calculated at each current frame/ result toward zero. 

picture (I, P or B) if the previous frame is an I or a P picture example, assume N^15 and M»3 initially. Then, M 

by checking the temporal references of the current picture changes to M-2. Using eqn. 8, the adjusted program GOP 

and the previous I or P picture. If the previous frame is a 35 length N;«14. 

B-picture, we do not check the sub^GOP. In MPEG syntax, 2. Second, if N; changes, we reset the super GOP length 

the temporal reference of a picture is included in the picture t»y ^i^g equation (1). The new super GOP 540 now starts 

header that is located at the beginning of the segment of ^ight at the following picture, at time 520. Note that the 

compressed bits for the picture. Specifically, the term program GOPs do not have to be aUgned with the new super 

"temporal_reference" is a 10-bit unsigned integer associ- 4o QQp^ -j^^ nMmhtx of I, P and B pictures in the (new) super 

ated with each input picture. It is incremented by one, ^OP 540 can also be calculated with the new super GOP 

modulo 1024, for each input frame. For example, temporal. length, say N„,^, as, respectively, n,^N,/(N/N,)oN/N;, 

reference may have values of 67. 68, 69 and so forth. Hence, n, p-N, p*(N/N;)-N/M;-n; . and n, «-N, „*(N/N^-N-n; p- 

a transcoder (or, more specifically, the associated GOP jj 

structure monitors 142 or 152 of FIG, 1) can access the 45 \ pi^ally, the nominal number of bits for the (new) super 

temporal reference of a current picture before the transcoder qqP is now equal to: 
actually processes (transcodes) the picture. The temporal 

reference of the previous I or P picture is already available Ti^nN^'i^^n^y^chann^ipp/y {^) 
because the picture has been processed. 

FIG. 5 illustrates a video program with a change in the so Th« ^^^^^^ ^f bits for the first (new) super GOP 540, 

sub_GOP length, and subsequent adjustment of the super however, may be slightly different than for other, subsequent 

GOP length, in accordance with the present invention. super GOPs that follow the super GOP 540 since the 

Note that the program GOPs for the different programs previous super GOP 530 (with length J is an unfinished 

1, . . . , Ldo not necessarily align with one another. ^^^P^' that has leftover bits that were previously 

An example program I that is transcoded includes GOPs 55 allocated, but not yet used. In accordance with the invention, 

546, 548 and 550 having a first GOP structure (GOP length ^^^^ ^an be aUocated to the first new super GOP 540. 

and/or sub_GOP length), and GOPs 552, 554 and 556 Specifically, let N^^be the old super GOP length and N' be 

having a second, different GOP structure. Note that the GOP ^1"°^*^^^ of frames of program 1 in the old super GOP that 

structure can change for more than one program at the same h^^e been processed. The remainmg number of bits for the 

time. The super GOP length and bit budget arc adjusted 60 ^^st new super GOP is then: 

accordingly. ^. ^ . R'T,.s^-*{t,^'^...^R)] (10) 

Moreover, as mentioned, the programs may comprise 

transcoded data only, or both transcoded and encoded data. Here, Tj^^ is the nominal number of bits for LxN' frames 

At a time indicated at 510, an old super GOP 530 with and (T^xiv^-R) is the number of bits used for the frames in 

length, N^y, is determined. A sub_GOP (the distance 65 the old {unfinished) super GOP 530. Hence, [T^,,^- 

between two P-pictures in either encoding or display order) (Tjr^xjv denotes the leftover bits from the old 

for each program does not change up until a time indicated (unfinisned) super GOP 530. 
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With the updated information, we can determine the target 
rates for the current and future frames (eqns. 3 and 4). Note 
that the sub_GOP length of a program can be verified and/or 
corrected at each frame if the previous frame is an I or a P 
picture, and if necessary, other parameters are also adjusted 
accordingly (cqns. 8-10). 
3.2 Program GOP Length 

Let N, and N/^^^ be the GOP lengths of the previous 
and the current GOPs for program 1, respectively. There are 
three possible cases regarding the GOP lengths, i.e., 



For encoders, both N, and N,,^^ are available. 
However, for a transcoder, the information on GOP length is 
embedded in the input pre -compressed bit stream. Before 
completely processing the current GOP of program 1, N/^^ 
is not available. 

We set the current GOP length N;=N,^„,j for the 
transcoder. Clearly, if N; jurw^H^w* hdcvt a correct GOP 
length for program 1. Hence, no further action is needed. 

FIG. 6 shows an example where the new GOP length for 
the current GOP is longer than the currently used GOP 
length. 

Program 1 includes GOPs 646, 648 and 652 having a first 
GOP structure, and GOPs 654, 656 and 658 having a second, 
different GOP structure. 

An old super GOP 630 starts at a time 610, while a new 
super GOP 640 starts at a time 620. Old GOPs extend, e.g., 
from a time 632 to a time 635, and from a time 635 to a time 
620. A new GOP extends from a time 635 to a time 650. 

If N;,^^>N;=N;^^^ i.c., the actual (new) program GOP 
length is longer than the currently used (old) GOP length. 
The GOP structure monitor (142, 152) expects to reach a I 
picture at the end of the current GOP of N, frames, at time 
620. However, the end of a program GOP is not actually 
reached at time 620. In accordance with the invention, this 
alerts the GOP structure monitor of a change in the GOP 
structure. Specifically, it can be concluded that the actual 
GOP length is longer than the currently-assumed GOP 
length. 

Moreover, because of the nature of the program GOP 
structure (RG. 2), the new GOP length will be at least M; 
(one sub GOP length) additional pictures longer. Here, we 
have assumed that the sub-GOP length M does not change 
within a GOP. We extend the current GOP of N/ pictures by 
M, additional pictures, i.e., 



02) 



With the new N,, we recalculate the super GOP length by 
equation (1), and the nominal number of bits for the first new 
super GOP (and the following super GOPs) by equation (2). 
The first new super GOP 640 is aligned with the current P 
picture of program 1, as shown at time 620. Because of the 
leftover bits from the previous super GOP 630, the number 
of bits for the first new super GOP 640 is re- allocated as: 



(13) 



where N^,^ and N,^^ are the old and new super GOP lengths, 
respectively, and N' is the number of frames of program 1 in 
the old super GOP 630 that have been processed. If the 
extended GOP for program 1 is still shorter than the actual 
GOP, we repeat the above procedure (e.g., increasing N, by 
eqn. 12) until reaching the end of the actual GOP. 
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FIG. 7 shows an example where the new GOP length for 
the current GOP is shorter than the currently used GOP 
length. 

Program I includes GOPs 746, 748 and 752 having a first 

5 GOP structure, and GOPs 754, 756, 758 and 760 having a 
second, different GOP structure. 

An old super GOP 730 starts at a lime 710, while a new 
super GOP 740 starts at a time 720. An old GOP extends 
from a time 732 to a time 735, and from a time 735 to a time 

10 750. A new GOP extends from a time 735 to a time 720. 
If ^i^new^^r^Loidf actual GOP length is shorter 

than the currently used GOP length, an I picture (which 
denotes the start of a new GOP — there is only one I picture 
in a GOP) will be reached (at time 720) before the end of the 

15 current GOP of N/ frames. At this point, the GOP structure 
monitor knows the actual GOP length, N, since the GOP 
754 has just concluded. We set N^N/ „,^,. Moreover, with 
the new program GOP length, N, we reset the super GOP 
length by (1), and recalculate the nominal niunbcr of bits for 

20 the new super GOPs by (2). Again, because of the leftover 
bits from the previous (unfinished) super GOP 730, the 
number of bits for the first new super GOP 740 may be 
slightly different than for subsequent super GOPs, that is. 
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where N^/^ and N„^^ are the old and new super GOP lengths, 
respectively, and N' is the nimober of frames of program 1 in 
the old super GOP 730 that have been processed. 

4. RATE CONTROL 

Once the target rate for a frame of a program is set, the 
next step is to achieve this rate, e.g., by adjusting the coding 
parameters, such as the quantization parameter, in the encod- 
ers and transcoders of the stat mux/remux system. Any 
suitable rate control scheme, such as those for use with the 
MPEG encoder, can be used. Some of them can be applied 
to the transcoders as well. 

Preferably, joint rate control is used to dynamically dis- 
tribute the channel capacity among the programs according 
to the programs* relative complexity measures. 

5. CONCLUSION 

A Stat remux system is presented having encoders and 
transcoders that can handle both uncompressed digital video 
signals and pre -compressed video bit streams. The system 
implements a novel adaptive rate control, which dynami- 
cally distributes the channel capacity over the input video 
programs, either uncompressed digital video signals or 
pre<ompressed video bit streams, on a frame -by-frame 
basis. The bit allocation strategy adopted in the adaptive rate 
control is able to address changes in picture organization 
(e.g., changes in GOP length or sub_GOP length) in the 
input pre-compressed bit streams. When such a change is 
detected, the bit allocation is recalculated. 

Moreover, hierarchical bit allocation is provided on a 
super GOP level, then on a super frame level, and finally at 
an individual frame level. 

Although the invention has been described in connection 
with various specific implementations, it should be appre- 
ciated that various adaptations and modifications can be 
made thereto without departing from the scope of the 
invention as set forth in the claims. 

What is claimed is: 

1. A method for transcoding a plurality L of pre- 
compressed video programs, comprising the steps of: 
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estimating a respective GOP structure for each video 
program; 

allocating bits for transcoding the pre-compressed video 
programs according to a hierarchical scheme that 
includes at least: (a) a super group of pictures (GOP) 
level, wherein a super GOP comprises at least one GOP 
from each of the video programs, and (b) an individual 
frame level; 

wherein said allocating step is initially responsive to said of: 
estimated respective GOP structures of the video pro- 
grams; 

monitoring the actual respective GOP structures of the 
pre-compressed video programs to detect any changes 
thereof; and 

when a change in the GOP structure of at least one of the 
video programs is detected, adjusting the allocation of 
bits for transcoding the video programs in the super 
GOP according to said change. of: 

2. The method of claim 1, wherein: 
said monitoring step monitors the GOP structure of the 

video programs on a frame-by-frame basis. 

3. The method of claim 1, wherein: 
the change in GOP structure is due to a change in a video 

source. 

4. The method of claim 1, wherein: 
the change in GOP structure is due to a commercial 

insertion in the associated video program. 

5. The method of claim 1, wherein: 
an aggregate of the transmission rates for the video 

programs is maintained within a total available channel 
rate. 

6. The method of claim 1, wherein: 
the video programs are communicated over a broadband 

communication network to a decoder population. 

7. The method of claim 1, wherein: 
the hierarchical scheme includes a super frame level 

between the super GOP level and the frame level; and 
each super frame comprises L pictures at a common frame 
instant, one from each of the video programs. 

8. The method of claim 1, wherein: 
the L video programs have respective variable bit rates. 

9. The method of claim 1, comprising the further step of: 
determining whether a target bit rate that is set in said 

allocating step for a frame of pre-compressed data of at 
least one of the video programs is greater than a 
measured bit rate of the frame and, if so, reducing the 
target bit rate to the measured bit rate. 

10. The method of claim 1, wherein: 
the plurality of video programs further comprise at least 

one video program having uncompressed video data. 

11. The method of claim 1, wherein: 
the GOP structure comprises a program GOP length. 

12. The method of claim 11, comprising the further step 

of: 

starting a new super GOP, and allocating bits thereto, 
when a change in the length of a program GOP of at 
least one of the video programs is delected. 

13. The method of claim 11, wherein said estimating of a 
respective GOP structure for each video program comprises 
providing an assumed value for the program GOP length 
before a measurement of the program GOP length can be 
made; 

said method comprising the further steps of: 65 
measuring the program GOP length when the measure- 
ment can be made; and 
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comparing the mea^red program GOP length to said 
value and, if there is a difference, setting said value 
to the measured program GOP length. 

14. The method of claim 12, wherein: 
the allocation of bits to the new super GOP accounts for 

unused bits from a previous super GOP that were 
previously allocated thereto but not used. 

15. The method of claim 12, comprising the further step 



calculating a length of the new super GOP based on the 
changed program GOP length. 

16. The method of claim 14, wherein: 
the unused bits are determined based on a number of 

frames processed in the previous super GOP. 

17. The method of claim 1, wherein: 
the GOP structure comprises a sub GOP length. 

18. The method of claim 17, comprising the further step 



starting a new super GOP, and allocating bits thereto, 
when a change in the length of a sub GOP of at least one 
of the video programs is detected. 

19. The method of claim 17, wherein: 
the sub GOP length is calculated after bits are allocated in 

said allocating step to an I picture. 

20. The method of claim 17, wherein: 
the sub GOP length is associated with a distance between 

one of successive I-pictures or successive P-pictures in 
a program GOP. 

21. The method of claim 17, wherein: 
when a current picture of one of the video programs is an 

I or P picture, and a previous pictm:e is an I or P picture, 
the sub GOP length is determined according to a 
difference between temporal references of the current 
picture and the previous picture. 

22. The method of claim 17, wherein: 
when a current picture of one of the video programs is a 

B picture, and a previous picture is an I or P picture, the 
sub GOP length is determined according to a difference 
between temporal references of the current picture and 
the previous picture, plus one picture. 

23. The method of claim 17, comprising the further step 
of: 

calculating the sub GOP length after processing an I or P 
picture. 

24. The method of claim 17, wherein said estimating of a 
respective GOP structure for each video program comprises 
providing an assiuned value for the sub GOP length before 
a measurement of the sub GOP length can be made; 

said method comprising the further steps of: 

measuring the sub GOP length when the measurement 

can be made; and 
comparing the measured sub GOP length to said value 
and, if there is a difference, setting said value to the 
measured sub GOP length. 

25. The method of claim 17, wherein: 
when a change in the sub GOP length is detected, calcu- 
lating an associated new program GOP length. 

26. The method of claim 25, wherein: 
the new program GOP length N;^^ is calculated accord- 

where is the previous program GOP length; 



is the new sub GOP length; and 



'*r denotes integer division with truncation of the result 
toward zero. 
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27. TTie method of claim 25, wherein: 

after the new program GOP length is calculated, calcu- 
lating a new super GOP length in accordance therewith, 
and allocating bits to the new super GOP accordingly. 

28. The method of claim 1, wherein: 
the GOP structure denotes picture types and dieir arrange- 
ment in a program GOP. 

29. The method of claim 1, wherein: 
a length of the super GOP is determined according to a 

least common multiple of the program GOP lengths of 
the respective video programs. 

30. An apparatus for transcoding a plurality L of pre- 
compressed video programs, comprising: 

means for inputting an estimated respective GOP struc- 
ture for each video program; 

means for allocating bits for transcoding the pre- 
compressed video programs according to a hierarchical 
scheme that includes at least: (a) a super group of 
pictures (GOP) level, wherein a super GOP comprises 
at least one GOP from each of the video programs, and 
(b) an individual frame level; 

wherein said allocating means is initially responsive to 
said estimated respective GOP structures of the video 
programs; 

means for monitoring the actual respective GOP struc- 
tures of the pre-compressed video programs to detect 
any changes thereof; and 

means for adjusting the allocation of bits for transcoding 
the video programs in the super GOP, when a change in 
the GOP structure of at least one of the video programs 
is detected, according to said change. 

31. An apparatus for transcoding a plurality L of pre- 
compressed video programs, comprising: 

a user interface for inputting an estimated respective GOP 
stmcture for each video program; 

at least two transcoders for receiving pre-compressed data 
of corresponding ones of the pre-compressed video 
programs; *o 

a rate control processor for allocating bits for transcoding 
the video programs according to a hierarchical scheme 
that includes at least: (a) a super group of pictures 
(GOP) level, wherein a super GOP comprises at least 
one GOP from each of the video programs, and (b) an 
individual frame level; 
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wherein said rate control processor is initially responsive 
to said estimated respective GOP structures of the video 
programs; and 

a GOP structure monitor associated with each of the 
transcoders for monitoring the actual respective GOP 
structures of the associated pre-compressed video pro- 
grams to detect any changes thereof; 

wherein said rale control processor adjusts the allocation 
of bits for transcoding the video programs in the super 
GOP, when a change in the GOP structure of at least 
one of the video programs is detected, according to said 
change. 

32. A method in accordance with claim 1, comprising the 
further steps of: 

transcoding of said pre-compressed video programs; and 
encoding a plurality of uncompressed video programs; 
wherein said allocating step includes allocating bits for 
encoding of said uncompressed video programs in 
addition to allocating bits for transcoding of said pre- 
compressed video programs based on said hierarchical . 
scheme. 

33. An apparatus in accordance with claim 30, further 
comprising: 

means for transcoding said pre-compressed video pro- 
grams; and 

means for encoding a plurality of uncompressed video 
programs; 

wherein said allocating means allocates bits for encoding 
of said uncompressed video programs in addition to 
allocating bits for transcoding of said pre-compressed 
video programs based on said hierarchical scheme. 

34. An apparatus in accordance with claim 32, further 
comprising: 

at least one encoder for receiving uncompressed video 
programs; 

wherein said rate control processor allocates bits for 
encoding of said uncompressed video programs in 
addition to allocating bits for transcoding of said pre- 
compressed video programs based on said hierarchical 
scheme. 
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